Topic-aware Contextualized Transformers
Authors: Ruiying Lu
Bo Chen
Dan dan Guo
Dongsheng Wang
Mingyuan Zhou
Venue:  (2021)

Abstract: Training on disjoint fixed-length segments, Transformers successfully transform static word embeddings into contextualized word representations. However, they often restrict the context of a token to the segment it resides in and hence neglect the flow of contextual information across segments, failing to capture longer-term dependencies beyond the predefined segment length. This paper uses a probabilistic deep topic model to provide contextualized embeddings at both the token and segment levels. It also introduces topic self-attention and a contextual next-word embedding guided topic select-attention, injecting contextualized topic information into Transformer-based architectures. Moving beyond conventional Transformers that ignore longer-range word dependencies and contextualize their word representations at the segment level, the proposed method not only captures global semantic coherence of all segments and global word concurrence patterns, but also enriches the representation of each token by adapting it to its local context, which is not limited to the segment it resides in and can be flexibly defined according to the task. Experiments on various corpora show that adding only a few extra parameters, the proposed topic-aware contextualized transformers consistently outperform their conventional counterparts, and can be used to generate coherent sentences and paragraphs.

Under review as a conference paper at ICLR 2021
TOPIC-AWARE CONTEXTUALIZED TRANSFORMERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Training on disjoint fixed-length segments, Transformers successfully transform static word embeddings into contextualized word representations. However, they often restrict the context of token the segment resides in and hence neglect the flow of contextual information across segments, failing to capture longer-tenn term dependencies beyond the predefined segment length. This paper uses a probabilis- tic deep topic model to provide contextualized embeddings at both the token and segment levels. It also introduces a contextual next-word embedding guided topic attention module, injecting contextualized topic information into Transformer- based architectures. The proposed method not only captures global semantic coherence of all segments and word concurrence patterns, but also enriches the rep- resentation of each token by adapting it to its local context, which goes beyond the segment resides in and can be flexibly defined according to the target task while maintaining control over memory footprint and computational time. Experiments various corpora show that adding only few extra parameters, the proposed topic-aware contextualized transformers consistently outperform their conventional counterparts, and can be used to generate coherent sentences and paragraphs.
INTRODUCTION
Language models (LMs) play an important role across a range of natural language processing tasks, such text summarization (Rush al.) 2013 Gehrmann al. 2018), neural machine translation
(NMT) (Sutskever et al., 2014 Cho et al., 2014 and image captioning (Herdade et al.) 2019 Anderson et al.) 2018 Xu al. 2015). Existing neural LMs are often built on either recurrent units, as used in recurrent neural networks (RNNs) (Cho et al. 2014b) Hochreiter and Schmidhuber 1997). or purely the attention mechanism based modules, as used in the Transformer and its various generalizations (Vaswani et al. 2017] Dai et al., 2019 Radford et al.) 2019). Moving beyond traditional recurrent units, Transformers mainly rely on attention mechanisms, in which the direct connections between long-distance word pairs might ease optimization and enable the learning of long- range dependency (Dai et al. 2019), and have recently demonstrated state-of-the-art performances on wide range of sequence modeling tasks.
Rather than representing a token using a predefined word embedding vector, each Transformer layer creates contextualized representation of each token by attending to different parts of the input segment (Fthayarajh) 2019), allowing the same word to take different representations depending on its context. However, Transformers are usually trained on disjoint fixed-length segments, without any information flow across segments (Dai et al. 2019), limiting the contextualization within the current segment. Therefore, they often fail to take full advantage of many other rich contextual information, such as longer-range word dependencies beyond the segment length and semantic relationships between neighboring segments. While naive solution explore richer contextual information is to increase the segment length, in practice, it is usually infeasible due to limited resources, which requires O (N2) for the window N of inputs each layer.
Some long-range transformer variants (Dai et al. 2019: Rae et al. 2020, Rae and Razavi 2020 aim to extend context via compression, which use compressed memory cells for preserving the previous segments' information. The Transformer-XL (Dai et al. 2019) builds up recurrent connections between segments, concatenating the past activations with a memory cell of size M 2 N, which results in an attention cost of O (M N)). However the memory cell still requires a considerable space L x M x d'maded in a L-layer transformer with embedding size of deoodets which consumes non-negligible cost (Rae and Razavi 2020).
Rae et al. (2020) shorten the range of attention

---

Under review as a conference paper at ICLR 2021
for Transformers by compressing the past memories into fine- and coarser compressed memory slots, while still suffering from memory consuming as the memory size is quiet large (> 1000). In addition, some efficient versions focusing on Transformer model's self-attention mechanism have also recently been explored. These models reduce memory requirements by leveraging sparsity in the attention layers (Sukhbaatar al. 20T9), exploiting factorized sparse representation(Chin et al.] 2019)) replacing dot-product attention with locality-sensitive hashing to decrease complexity (Kitaev et al. 2020), or using product-key attention increase the key space (Lample et al. 2019). Besides, Chen et al. (2019) represent sentence-level context as latent topic representations by using a convolution neural network, and utilize the context representations to improve translation. However, leveraging the contextualized topic information by capturing semantic coherence via a deep probabilistic topic model, to our knowledge, has not been directly applied to Transformer before. Furthermore, compared with pre-training, fine-tuning is relatively inexpensive (Devlin et al. 2019). Nevertheless, most of the current contextualized models are trained independently on different datasets, without making good use of the publicly released pre-trained
2019 Devlin et al. 2019 Radford et al.. 2018 Brown et al., 2020) Petees et al. 2018 Yang al.,
models (Radford et al.
large amount of training data. This motivates us to explore general intervention based on those predecessors for performance gain with little computation cost, providing longer-range dependencies through a deep topic model.
2019 paired with unsupervised pre-training
Different from RNN or Transformer-based LMs, topic models (Blei et al. 2003 Teh et al. 2006 /hou
and Carin 2015 Gan et al. 2013| Zhou et al. 2016 Zhao et al.] 2018) are well suited for capturing
global semantic coherency by extracting word concurrence patterns into semantically meaningful topics, which can be viewed as the contextualized word representations of the entire target corpus including all segments. Since topic models are appropriate to capture long-range dependencies, some approaches attract significant recent interest by leveraging topic models to improve
language models (Dieng et al. 2017 Ahn et al., [0166] Lau et al. 2017 Wang et al. 2018a Guo et al.) 2019). Dieng et al. (2017) and Ahn et al. (2016) integrate the syntactic dependencies of RNNs and semantic topics of latent topic models. Lau et al. (2017) introduce an attention based convolutional neural network to extract semantic topics for extending the RNN cell. Wang et al. (2)8aa) learn the global semantic coherence of document via neural topic model and use the learned latent topics to build a mixture-of mixture-offeepprts language model. [uo et al. (2019) extract recurrent hierarchical semantic structure via dynamic deep topic model to guide natural language generation. Motivated by recent successes on integrating topic information into RNN-based LMs, here we focus on using topic model to provide richer contextual information for improving the Transformer. In particular, we consider using Poisson gamma belief network (PGBN) (Zhou et al. 2016 Zhang et al. 2018). state-of-the-an probabilistic topic model which can be equivalently represented as a multi- -tochastic-laye deep generalization of vanilla topic models (Blei et al.] (003)] /Zhou et al. 2012), to extract globally shared semantical topic representations of user-defined contexts.
To this end, three different types of contextual topic information are provided to introduce long-range (i) We first introduce the contextual token embedding (TE) guided by topic model to enrich the representation of each token, which not only extracts global semantics from the corpus, but also provides localized representation of a token given either its preceding or surrounding context (which one to use is task-dependent). (ii) To utilize contextual
semantic dependencies into Transformers.
information of a segment, we develop the contextual segment embedding (SE) to construct a set of virtual words, which is placed before the word sequence of the current segment and fed into Transformer. As such, the generation of any token in one segment depends on semantic context from the previous segments. (iii) After that, we further develop multi-head topic attention (TA) module into the Transformer, selecting semantically related topics for generating each token, a design inspired by how a token generated by a topic model given the topics and corresponding topic proportion. To encourage topic select-attention to focus on the topics where the predicting token is more likely to be assigned to by the topic model, during training, we add a restriction between the attention weights and the latent representation of the predicting word. Besides, sparse penalty is employed on the topic select-attention, encouraging the network to focus on only small subset of extracted topics. Moving beyond conventional transformers, our model can not only utilize longer-range word dependencies beyond the segment length and semantic relationships across all segments, but also generalize easily to any pre-trained Transformer- based model by jointly fine-tuning on the target corpus. It only adds minor memory and computation overhead comparing with fine-tuning the Transformer-bate model alone. We demonstrate the effectiveness of our method both quantitatively and qualitatively.
2

---

Under review as a conference paper at ICLR 2021
2 PRELIMINARIES
To train Transformer-based LMs, the underlying word sequence of a corpus is usually broken into fixed-length non-overlapping segments, without any information flow across segments. We denote 8, = (s.1)...., (n1) as the n-th segment of / consecutive tokens, where Sni E {l,..., ... V} and V is the vocabulary size. The segment length is chosen to balance the ability model long-range word dependencies with the memory and computation cost. Note the segments fed into Transformers no longer respect natural document boundaries, which means a segment could consist of the tokens from more than one document. Below we provide brief overview of Transformers (Vaswani et al. 2017) and PGBN (hou et al. 2016). a multt-stochastic-aayee -laee deep topic model. To make them compatible with each other, each segment is fed into Transformer document analyzed in PGBN.
Vanilla Transformer networks: Like a standard LM, Transformers are trained by maximizing the likelihood of all segments L Σn [(s,), [(s,) E, log Pq (smi 8n,<s) where <n consists of the preceding tokens of Sni within the nah segment, and S the parameters for modeling the conditional probability. Our proposed method can be applied to improve both Transformer encoder and decoder architectures (Vaswani et al. 2017: Dai et al. 2019, Radford et al. 2019 Devlin
al. 2019 Radford et al. 2018). For brevity, will mainly show how to PGBN to better contextualize through the Transformer decoder, which consists of L layers as
2 = WE+ + PE, z' = TransformerBlock (2'-1), P(u) = softmax WZ)
(I)
where WE and PF are the word and position embeddings of 8n,i-1 when predicting the i-th token of the segment, W. E RV the embedding matrix, and Z I. the outputs of all L layers, each of which consists of multi-head self-attention block followed by a feed-forward neural network (Vaswani et al. 2017). To facilitate these connections, all layers the model, as well as the embedding layers, produce outputs of dimension doodel- See previous works (Vaswani et al. 2017: Devlin et al. 2019) for more details on Transformers.
Deep topic model: PGBN is used to provide semantically meaningful contextual representations to guide Transformers. We represent segment s, as a bag-of- (BoW) count vector d. E zY,, the v-th element of which counts how many times term in the vocabulary of size V appears at the n-th segment. The generative model of PGBN with T hidden layers, from top to bottom, is expressed as
Gamma (r. T T-1) 0% ~ Gamma ( +(1)
Gamma @0 Th) d,, Poisson 4 O!))
where the shape parameters of gamma distributed hidden units 0 E TR are factorized into the product of connection weight matrix gz+1 E R Mex and hidden units 0%+ of the next layer. The global semantics of entire training corpus are compressed into 1: representing topic relations of layers. 0% denotes local semantic representation of input d,,, indicating its topic proportion 1-th layers. See /hoo et al. (2016) for more details on PGBN.
3 CONTEXTUALIZED TRANSFORMERS
In Transformer-based model, an essential step is to introduce word embedding matrix W. V Vxdmosi the w-th row of which provides a Senode--dimenional representation of the v-th token of the vocabulary. This matrix is often pre- pre-trained large corpora and fine-tuned afterwards target corpus, where each token is simply represented with its corresponding embedding vector in W.. Given W.. the Transformer architecture itself can be considered transforming each input segment, represented as a sequence of static word embedding vectors, into a sequence of contextualized word representations (Ethayarajh) 2019). which allow the same word take different representations depending on its context. However, the contextualization is often limited to the segment itself of a fixed length, neglecting the longer-range word dependencies beyond segment length and semantic relationships between neighboring segments. To advance the longer-context information, we consider providing richer contextual information to guide the Transformer with PGBN, which is good at extracting globally semantic topics and localized feature representation of context. Fig. (a) shows the overall architecture of the proposed model, where a basic Transformer block is in conjunction with multi-layer topic model. Firstly, the topic model extracts the contextual representation of each token as TE, directly adding to the embedding space, and the contextual representation on segment level as SE, which is placed in front of the current segment. Then an additional multi-head topic
3

---

Under review as a conference paper at ICLR 2021
Figure (a) The over- all architecture of the proposed model, where the TE, SE and TA mod- ules are highlighted in yellow color. (b) Visu- alization of the proposed contextual token embed- ding (TE) for token where WE and denote word embedding and po- sition embedding. (c) Visualization of segment embedding (SE), analo- gized as virtual tokens and placed before the word sequence. (d) Vi- sualization of the topic select-attention module, interleaved into trans- formers as shown in (a).
Sottmax
WE
(a) architecture
Linear
entropy
Norm
LuM
P
Contertual Topic Attention
Feed Forward
spane
Norm
topic
Contestual Token Embeddding
Topic
guačance
Select Attentin
Se
Ri
Name
Maskeal muli-hzad
sinale
Attention
(c) Contextual Segment Embedding
Topic Model 18
6
q
6
0O0000
q 02
6
1ooo
WE
(d) Topic Select-Afteation
attention module is used attend different semantic topics as keys and values, according the query calculated from the Transformers. Furthermore, the sparse regularization and next-word prediction guidance are employed.
3.1 MULTI-LAYER TOPIC-AWARE CONTEXTUALIZATION
In this section, we will show how integrate the hierarchical topic semanties learned with T- stochastic-layer PGBN 2 into existing Transformer-based models. The topic matrix of I-th layer extracted from the target corpus is represented as &' = II:.,, &x E B containing M, topics, which tend be more semantically specific lower layers and become more general when moving upwards. The normalized topic proportion vector 0¹ (01....,04,) over v summarizes the representation of the local context of a token. Besides, we define 0' over v as the contextual representation at the segment-level, which represents the first several segments preceding the current one, shared within the current segment.
Oh,)'
Contextual token embedding (TE): To enrich the representation of each token by adapting it to both the target corpus and the local context it resides our first key idea is introduce contextual token embedding guided by multi-stochasttc-layer topic model. More specifically, we first need to define the local context of token depending on the learning task, which composed of its preceding tokens (e.g. for text generation) or both its preceding and following ones (c.g. for text classification), going beyond the current segment u resides in. We summarize into BoW vector d z. As shown in Fig (b), given the topic matrix T shared globally by all segments, we use an inference network to project d to that represents the topic proportions of d under v We then define localized contextual feature vector as vf. O (0") E R where bb. RR-, denotes u-th row of v¹ and denotes an element-wise product. other words, the local context topic proportion 0² used to re-weight topic vector " Thus the m-th element of vt. @ (0') O (0')ll represents the probability of assigning token to topic m layer t. Since topics at different layers reveal hierarchical aspects of the context, we fuse the topic information from all layers together the contextual token embedding vector, expressed as
user-defined local context of
TE Σ
-t (ubt
(0') @ (0'YlliJWie.
a projection matrix mapping the M,-dimensional feature vector to a dimennsainnl contextual token embedding vector. Note the contextualized token embedding depends on not only its position in the vocabulary, but also its local context that determines the proportions of different topies, which reflect the underlying semantics of the local context. For each token, we modify its fixed embedding vector from W. W. by adding its topic-guided contextual embedding and the position embedding, i.e., E WE+ TE PF.
where Whe E MRM

---

Under review as a conference paper at ICLR 2021
Contextual segment embedding (SE): The second key idea of the paper is to provide a localized representation of each segment given its local context, which is defined as the first several preceding segments for text generation. More specifically, vector d E zV. Given the topic matrix Vp' at layer i, we infer the topic proportion vector of d z! as 0" which serves as a contextualized embedding vector for the current segment. As shown in Fig (c), the contextual segment embedding matrix with hierarchical topic information is constructed as
summarize these preceding segments into BoW
SE Concat[SE" ..... "E,,,..,SEE"" S." = ( diag((")))
(4)
where SE E Ii, Xdmoost denotes the segment embedding at layer t, each row of y is element- wisely reweighted by Ø', which is then further projected by W. into the input embedding space. Fach row of this contextual segment embedding matrix is considered as a localized embedding of a topic (i.e., a column in )²") and here analogized as a contextual virtual token. By concatenating segment embeddings across all layers, SE contains >x= M, virtual tokens and can be placed before the word sequence of the current segment, shown in Fig I (a). In order to distinguish the virtual token from original input token, we further add embedding (EA, Eg E [Rdmadee) respectively to help discriminate them. As these virtual words are not ordered, their embedding vectors are not combined with any position embedding. By integrating the contextual semantic information into the input space of Transformer by those virtual tokens, all real tokens in the original segment can relate to all topics with standard self-attention in the Transformer blocks. More specifically, each token in the segment is accessible to all the virtual tokens (segment embeddings) from specific to general perspective.
Topic Attention (TA): The third key idea of this paper is to add topic attention (TA) into Transformer layers, which is implemented with a topic select-attention block. As shown in Fig [ (a), for each layer of all T topic layers, the topic matrix To E RR X (containing M, topics) is first projected through the word embedding matrix We. reducing the dimension of each topic from V to dmodel- Then the projected topic vectors are then fed into a layerorm layer, following the implementation of Vaswani et al. (017), calculated as
LayerNorm (T') WW. E R Mr X dmoode
(5)
Then we build a multi-head topic select-attention to explore the relation between the query q, of standard Transformer and the M, topics V which is desired to select semantically related topics given a token. As shown in Fig. [ (d). with transforming matrices W W E Rddmddd x Xdmoddt this V can be projected as keys K vi 'W R.Mc Xdmadel and values V + w' E R. Thus, we attend q; into M, keys to obtain attention weights as
a softmax(y, (K*) //V////aa dmoota) E R
(6)
which are then used to aggregate the values into topic attention output as a, V E R modded This provides a natural way to leverage global semantic topics into Transformers.
(i) Regularization of contextual next-word embedding: For language generation, a common goal of learning the attention output of a, in ( is to better predict token u, given previous tokens Ugi. Note
Pi 6* Hence, in order to guide the topic select-attention with next-word embedding, we can regularize the attention weights with a loss function as pipriedet = ||a, 11. where = (pp.,,,..,PP.d)) ......) and the indices of heads and layers are omitted for brevity. Intuitively, we want query q; to attend on topics where the predicting token 2n is more likely to be assigned to. In addition, the attention weight vector a; also encouraged be sparse with L1-norm as Li,eparse = ||a:||| / /||a.]/2. The intuition behind this regularization is that token is often only strongly associated with small subset of topics.
in topic model, token u, chooses the m-th topic with probability Pt.m En 10 "eimm 6
(it) Integration TM into LM: Considering the multi-layer topics, it reasonable to integrate them into Transformers in a hierarchical way. As Rae and Razavi (2000) remark that it is not necessary to use long-range memories at each model layer, placing them in the latter layers and interleaved across the network with equal spacing result in good performances. Sukhbaatar et al. (2019) also observe that transformers converge to using smaller attention spans for lower layers in the network, which is corresponding to the concrete topics of bottom topic layers. Thus we interleave the multi-layer topics into transformers with equal spacing from the bottom to up layers. Take a three-layer topic model as an example, the topics from layers 1, 2, 3 are integrated into layers 4, 8, 12 of the transformer (12
5

---

Under review as a conference paper at ICLR 2021
layers in total), respectively, through the topic select-attention module. In other words, the query lower layers of Transformer attends to more specific topics captured by the bottom layer of PGBN, and the query at higher layers focuses those general topies from upper layers.
Based on the proposed contextualized token and segment embeddings and a novel topic attention module, construct topic-aware contextualized Transformers under multi-layer topic model to capture longer-range word dependencies beyond the segment length and semantic relationships between neighboring segments. Note those three modules can extend as long context as you wish, which can be set flexibly depending on tasks, without more computation consume. Afterwards, we will describe how to jointly fine-tune the contextualized transformers with those topic interventions.
3.2 MODEL INFERENCE
The proposed contextualized Transformer learns topic model and language model jointly, whose loss functions are denoted as LT M and LLM. respectively. For training PGBN, all segments of the target corpus are treated as BoW vectors (d1 ..., dy) ), ignoring word order. We introduce a Weibull hybrid autoencoding inference (WHAI) network (encoder) (Zhang et al. 2018) for PGBN (decoder). Denoting Q
II II ((0), dn)), the negative ELBO of PGBN can be expressed as
T
(0% dn) P ( + 0
LTM = Σ Eo P (dn T 0')] +
Σ>
Ee In
(7)
where the weight matrices (0')7= are updated with SG-MCMC in Cong et al. 2017), and the parameters of the inference network are denoted as W, By integrating the proposed topic information into an existing Transformer-based LM and adding the restrictions on attention weights a, the loss LLM over set of training examples U = {ui,..., ur} } is defined as
LLM E P (u:luce St ( 0², 0T -L, Lisprradeet::
where represents the Transformer-relateer parameters. Therefore, the final training objective is linear combination of LTM from the PGBN based topic model and LLM from the Transformer-basee LM, which minimized estimate {52, W,, (e'))==) More details are included in Appendix A
4 EXPERIMENTAL RESULTS
We first provide quantitative comparisons on two different natural language processing tasks, and then qualitative analysis to illustrate how the proposed contextualizations help improve Transformers. To verify the effectiveness of our method, we integrate topic-based token embedding (TE), segment embedding (SE), and topic attention (TA) into existing pre-trained Transformer models, fine-tuning from the released checkpoints. For each task, our method shares the same model architecture as the baseline. All models are optimized and evaluated on single 2080Ti GPU within a few hours. We use the Adam optimizer (Kingma and Ba 2015), where the experimental settings remain the same as baseline models provided by the authors. We use [M1, M2, M3] = [100, 80, 50) as the number of topics in three-layer PGBN, and set the dyper-parameters provided in the Supplement.
1, 1, 1. Python code is
4.1 QUANTITATIVE COMPARISON
Language generation We choose GPT-2 (Rafford et al. 2019) and Transformer-XX Dai et al. 2019) as baseline LMs. GPT-2 is realized by pre-training Transformer decoder and then fine-tuning on each specific task. Transformer-XX introduces a segment-level recurrence mechanism to learn dependency beyond fixed length without disrupting temporal coherence. We use perplexity as the evaluation metric and consider three publicly available corpora, including WikiText-1033 (WT103) Merity al. 2017). WikiText2 (WT2) (Merity et al., 2017). and Penn Treebank (PTB) (Mikolov and Zweig 2012). WT103 and WT2 contain 103M and 2M training tokens from Wikipedia articles. respectively, y, and word-level PTB has only 1M training tokens. Given the pre-trained GPT-2, we fine-tune contextualized GPT-2 on each of these three datasets, with the same vocabulary, tokenizer and experimenting settings as used in GPT-2. Different from GPT-2, Transformer-X XL is trained on each dataset respectively, where the authors only provide pre-trained model on WT103 but not on PTB and WT2. Since WT103 is the largest available word-level language modeling benchmark with long-term dependency (Dai al. 2019, it is feasible to use the pre- pre-trained model on WT103 as baseline to fine-tune on three datasets. Both the preceding segment window sizes TE and SE are set 4 for text generation. We conduct ablation studies to examine the effects of three proposed

---

Under review as a conference paper at ICLR 2021
Table Perplexity of different models (lower is better )
GPT-2-base
Transformer- -L--are
Model
WT103
WT2
# Param
WT103
WT2
buseline + line-tume
117M
16.33
14.66
15.22
257M
18.30
17.86
33.71
Token embedding (TE)
117+5.24M
16.15
13.98
15.08
257+1.07M
17.87
16.26
32.67
+Segment embedding (SF)
117+5.07M
16.10
13.92
14.98
257+0.83M
17.90
16.83
32.65
+ TTpic atlention (TA)
117+5.81M
16.01
13.92
15.00
25742.07M
17.86
16.30
32.64
+ TE SE TA
117+5.98M
15.82
13.67
14,92
25742.31M
17.84
16.23
32.60
Perplexity
22
GPT-2 + SE
GPT-2 TE
GPT-2
Perplexity
25
T-XL
T-XL
L-predict
18
GPT-2 ALL
GPT-2
20
T-XL TA
T-XL ALL
L-sparse
Basement
14
90 100
15
Topics
layer1
2000
4000
600D
0
2000
4000
6000
8000
(a) Time(s)
(b) Time(s)
Attention weights
Figure 2: (a) (b) Comparisons of test perplexity as function of fine- tuning time on WT2 based on GPT-2 and
Transformer-X XL (T-XL). (c) Visualizing of attention weights with different regularization.
modules: TE, SE and TA. As shown in Table D all three contextualization techniques improve both GPT-2 (BPE teken-level perplexity) and Transformer-Xi (word-level perplexity), combining the three techniques together leads to the best performance for both GPT-2 and Transformer- XL, only adding slightly more parameters. More evaluation on model varieties are shown in Appendix B
We further display in Fig. (b) how GPT-2, Transformer- XL, and their contextualized versions behave during fine-tuning, by showing the perplexity the WT2 test set over time. Obviously, while GPT-2 and Transformer XL behave well during the early stage of training, both of them show a clear trend of overfitting the training progresses. This overfitting trend especially concerning in Transformer-Xi although it is designed to utilize the contextual information across segments. Using the proposed contextualization methods, it takes much less time fit data well and exhibit strong resistance against overfitting.
The General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 201 8b) a collection of diverse natural language understanding tasks. To validate the efficiency of integrating semantic topics, we finetune the pre-trained 12 12-layer-Ber model on each dataset. For Glue tasks, we integrate topic semantics extracted from each input sentence. Thus SE and the next-word guided regularization Lpprdict can be neglected. We use batch sizes E {16, 32 fine-tune for 10 epochs and perform early stopping based on each task's evaluation metric on the dev set. The rest parameters remain the same pre-training
GLUE
Table 2: GLUE Development and Test results, scored by the evaluation server.
Data
System
MNLI-(m/mm)
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE ace 2.5k
Average
acc 392k
363k
acc 108k
acc 67%
pe 5.7k
F1 3.5k
Bert-base
84,7/83.4
70.2
88.4
93.0
52.3
78.7
86.5
66.4
78.2
Dev
+TE+TA
84,3/83.6
70.6
88.7
93.6
$2.3
79.0
87.5
68.2
78.6
Test
Bert-base
84.5/83.4
69.6
90.4
93.4
52.1
83.1
88.9
66.4
79.1
+TE+TA
84.6/83.8
70.0
90.8
93.7
53.1
83.8
89.1
79.8
It is clear that adding token embedding (TE) and topic attention (TA) into Bert outperforms baseline on different tasks, especially on small datasets. Take the RTE (2.5k) for example, there is 3.2% accuracy improvement over baseline, alleviating instability on small datasets. word, the tasks of GLUE benefit from the contextualized Transformer architecture, extracting globally shared semantic topics of input sentences and localized feature representations for each token.
Both the dev and test results are shown in Table 2
4.2 QUALITATIVE ANALYSIS
Efficiency of the regularization To verify the efficiency of our proposed regularization, we visual- ize the topic attention weights of randomly sampled word with different regularization terms. Shown in Fig. 2 (c), compared with the unconstrained example (the bottom row), sparse regularization leads sparsity into the attention vector. We also find the learned attention vector would focus on some formerly unnoticed topics by applying next-word topic guidance regularization. In the top row, it clear that the learned attention vector attend to the topics related with the predicted word and its context, while preserving sparsity. This underscores the effectiveness of our introduced regularization.
7

---

Under review as a conference paper at ICLR 2021
Tople-Layer1 investment director university course situation live values contributions futures
Topie-Layer2: securities director investors trust office marketing receive worth live Toplo-Layar million securitics director investors franc irust year carnings marketing cents
Generated sequences guided by toples:
A good place to invest is anywhere when you have more money. It usually comes to being able to use all the tools available to you, whether it's your skills, your experience, your esperience your skills for long term investment. When you have enough money, investing can help you to lot. You must be able to spend the money the best you can, do that, even if your invesiment will have to be few times more important than your ahility to reinvest. Most investors believe that they are at their greatest power once investments become mach smaller, however that opinion wrong because at best you're able to invest when you're just starting out, Sometines that power very strong, and at worst just docse't exist.
-
Preceding context: An anarchist was arrested last week on charges of atteemppingg to commit murder of three people in a crowded bar in Cambridge, recent days have prompted an investigation into incident.
England. munder sparked widespread ournge and protests
Generated sequences given precoding context:
It thought that the gunman who stabbed three people and injured two otherss was radicalised by the group after attending an anti-capitlist rally in Cambridge a week proviously. Police and campaigners say there is moumiing evidcace that he had links to extremist Islam and had travelled to the Middle East before joining the extremist movement. Two of the victims had been friends Mr Corbyn and had been invited join demonstrations to demand the end of British. Mr Corhyn suid he was not present at the time of the event, but believed to have been at the scene, Police are investigating the motive and motive of the man, said to his early 20s, who allegodly tried to commit murder.
Figure 3:
The generated sequences guided by multi-layer topics and preceding context. Words in the same color are semantic-consisten The generated texts successfully capture both syntactic and semantics.
Text generation given topics or preceding context Given the learned contextualized Transformer based on GPT-2, we can sample sentences conditioning on the topics from different layers. Shown in Fig the generated sequences are guided by a combination of topics at different layers, which is highly related to the given topies in terms of their semantics. These observations indicate that the con- textualized Transformer has successfully captured syntax and semantics simultaneously for language generation. Besides, we visualize the generated sentences conditioning on the preceding context, by integrating the encoded hierarchical topic representations of preceding context. Interestingly, we find the generated sentences successfully capture semantics and generate semantically-related words, which may not exist in the original document (highlighted with the same color). This phenomenon is also observed in Guo et al. 2019), which might be attributed to the introduction of semantic topics. More generated samples with longer preceding context are provided in the Appendix F where find our proposed model can memorize longer-range context than baselines.
Topic attention between words and topics To further illustrate the relationship between a word and its selected topics within the topic attention module, Fig. 4 takes the word "market" as example and visualize its attended topics of 12 heads at different layers. At each layer, we find the word is aligned to different topies, where each attention head potentially focuses different aspects of the input word. Specifically, the attended topics of "market" are semantically related to the 34-th topic ("billion, S, bank") and the 26-th topic ("data, technology, stock") at layer and so as in the upper layers, suggesting the efficiency of our proposed topic select-attention. In addition, we also find there are several heads attended the same topics, indicating those topics might be more helpful for predicting the target word. In other words, the regularization term of our topic select-attention module encourages the model to attend on its corresponding topic while keeping its variety.
MosBi-beads
65
62
Topic 47 23
25
26
34
40
52
76
82
92
95
44 15
22
23
44
46
49
50
51
61
71
79
13
8
17
22
25
26
27
28
34
40
42
44
R21 industry bill decision security technology bought stock Tapic H34 billion S tons bank commercial
84
sell computers food schools
business financial computer political #17 sook trading exchange investors
govemmment congress declined manager
Topic 246 foderal beads law naction
Tople N27 tax international financing expenses
Layer #76 export ban digital interattional
Layer A50 plan force order pound
Layer H40 econcemic oil sseets takeover
managernent contrals Support
2 companies banks invest organization
3 842
insurance industry bealth.
#92 goal failure commuuiity engineers
#7/ institute regime opartheid moral
H44 corp trade work nation
Visualizing the attended topics of word "market" at layers 1, and 3, respectively, X-axis denotes the index of attended topics, and y-axis the index of heads. Each row denotes the most activate topic of corresponding head and we omit the other inactivate topics. Several top words of corresponding topics are listed at the bottom.
Figure 4:
CONCLUSION
We introduce contextualized embeddings at both the token and segment levels to enrich longer- term dependencies beyond the fixed segment and semantic relationships across all segments of Transformer-based language models. Furthermore, to inject contextualized topic information into attention mechanism of Transformer-based architectures, a novel topic attention module, is further introduced. Experiments conducted on publicly available corpora demonstrate that the proposed topic-aware transformers outperform their conventional counterparts, providing better contextualized word representations for downstream tasks, and can generate coherent sentences and paragraphs conditioned on the designated multi-layer topies or preceding context.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization, In EMNLP, page 379-389, 2015.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush, Bottom-up abstractive summarization, In EMNLP, pages 4098-4109, 2018.
Ilya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. In NeurIPS, page 3104-3112, 2014.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406 1078, 2014a.
Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. Image captioning: Transforming objects into words. NewrlPS, pages 11 11135-11145, 2019.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, pages 6077-6086, 2018.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICMI., volume 37, pages 2048-2057, 2015.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares Dzmitry Bahdanau, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014b.
S Hochreiter and J Schmidhuber. Long short-term memory. Neural Computation, 98)):1335-1780, 1997.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, page 5998-6008., 2017.
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In ACL, pages 2978- 2988, 2019.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IIE IINNLP-IJCNLP), pages 55-65 2019.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. 8th International Conference on Learning Representationss ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Jack W. Rae and Ali Razavi. Do transformers need deep long-range memory? In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10. 2020, pages 7524-7529, 2020.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. In Proceedings of the 57th Conference the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 331-335, 2019.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint /904. 10509, 2019.

---

Under review as a conference paper at ICLR 2021
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Guillaume Lample, Alexandre Sablayrolles, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large memory layers with product keys. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 4546-8557 2019.
Kehai Chen, Rui Wang. Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. Neural machine translation with sentence-level topic context. IEEE ACM Trans. Audio Speech Lang. Process., 27 112):1970-1984, 2019.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Pupers), pages 4171-4186, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-HILT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Pupers), pages 2227-2237, 2018.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurlPS, pages 5754-5764 2019.
David M. Blei, Andrew Y. Ng. Michael I. Jordan, and John Lafferty. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993-1022, 2003.
Yee Whye Teh, Michael I Jordan, Matthew Beal, and David M Blei. Hicrarchical dirichlet processes. Publications of the American Statistical Association, (576)))115566 -5811, 2006.
Mingyuan Zhou and Lawrence Carin. Negative binomial process count and mixture modeling. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 37(2):307- 320, 2015.
Zhe Gan, Changyou Chen, Ricardo Henao, David Carlson, and Lawrence Carin. Scalable deep poisson factor analysis for topic modeling. In ICML, page 1823-1832, 2015.
Mingyuan Zhou, Yulai Cong, and Bo Chen. Augmentable gamma belief networks. J. Mach. Learn. Res., 17(163):1-44, 2016.
He Zhao, Lan Du, Wray Buntine, and Mingyuan Zhou. Dirichlet belief networks for topic structure learning. NeurIPS, page 7955-7966, 2018.
Adji B Dieng, Chong Wang, Jianfeng Gao, and John Paisley. TopicRNN: A recurrent neural network with long-range semantic dependency. In ICLR, 2017.
Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. A neural knowledge language model. CoRR, abs/ wtls//60..000 003.00 2016.
10

---

Under review as a conference paper at ICLR 2021
Jey Han Lau, Timothy Baldwin, and Trevor Cohn. Topically driven neural language model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pages 355-365, 2017.
Wenlin Wang, Zhe Gan, Wenqi Wang, Dinghan Shen, Jiaji Huang, Wei Ping, Sanjeev Satheesh, and Lawrence Carin. Topic compositional neural language model. In AISTATS, page 356-365, 2018a.
Dandan Guo, Bo Chen, Ruiying Lu, and Mingyuan Zhou. Recurrent hierarchical topic-guided neural language models. arXiv preprint arXiv: 1912.0/312 2019.
Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAL: Weibull hybrid autoencoding inference for deep topic modeling. In ICLR, 2018.
Mingyuan Zhou, Lauren Hannah, David Dunson, and Lawrence Carin. Beta-negative binomial process and Poisson factor analysis. In Artificial Intelligence and Statistics, pages 1462-1471, 2012.
Yulai Cong, Bo Chen, Hongwei Liu, and Mingyuan Zhou Deep latent Dirichlet allocation with topic-layer-adaptive stochastic gradient Riemannian MCMC. In ICMI,, 2017.
Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SL77), pages 234-239, 2012.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint 1804 07441, 2018b.


Reviews:
[{"review_id": "b_LgHFqdXwD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper introduces a global topic model into the Transformer to enrich longer-term dependencies beyond the fixed training segment, including contextual token/segment embedding (TE/SE) and Topic Attention (TA). However, some components seem to be unnecessary: The function of “+TA+SE” is very trivial by comparing “+Topic attention (TA)” with “+TE + SE + TA” in Table 1. In addition, the experiments of GLUE in Table 2 are only conducted on “Bert-base” with marginal improvements.\n\n---------------------------------------\nStrength:\n\n1. This work is the first to integrate the contextualized topic information via a deep probabilistic topic model into the Transformer architecture.\n\n2. Three different types of contextual topic information are provided to capture long-range semantic dependencies into the Transformer models. \n\n3. Experiments on various corpora show the effectiveness of the proposed methods, with only a few extra parameters.\n\n---------------------------------------\nWeakness:\n\n1. The ablation study is weak. TA and SE seem redundant in Table 1 compared with TA. The results in Table2 are marginal, given that the BERT-base baseline is relatively sensitive to hyperparameters. More language models (maybe larger) should be tested. Moreover, in view of (a), TE may be redundant together with TA. Since the results in Table are based on TE+TA, there should be a TE+TA comparison in Table 1. \n\n2. “Topic attention between words and topics” in the analysis seems farfetched because the attended topics of “market” are more than 34-th and 24-th. There are more similar but less relevant topics like 82-th, 50-th, etc. According to the examples, the topic information could be related to the semantic similarity of words. There might be a risk that the topic information would have already been entailed in the pre-trained language models.\n\n3. No experiments show the effect of hyperparameters choices from the topic model, such as the number of layers of PGBN as well as the topic number of each layer.\n\n4. Since this work is motivated to capture longer-range dependencies, it is not clear how the topic information helps. A comparison of the model performance for different lengths of input sequences would be helpful.\n\n5. There is no comparison with other methods about long-range transformer variants, which focus on the transformer model’s self-attention mechanism mentioned in Line 3, Page2.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OIqI3GXNBSZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a novel LM architecture that combines a Transformer with a PGBN topic model, enabling the transformer model to make use of addtional context topic information. The PGBN extracts topic information from the input, which is then used to enrich the information that is available to the transformer.\nThey propose three different methods of incorporating this context information from the topic model into the transformer: Topic embedding vectors to add to each token, segment embeddings to summarize preceding segments, and topic attention.\nThe topic model intervention can be applied to available pretrained transformers without the need to retrain the models from scratch. Experiments using pretrained GPT-2 and BERT show that incorporating topic information outperforms the respective baseline transformers both in language modeling perplexity and on the GLUE benchmark.\n\nTo my knowledge this is a novel model design. The information extracted by the topic model is provided to the transformer in three different ways, ensuring the utilization of all levels of topic information. It allows for topic information to be drawn from both the previous and the current segments of the input data.\nMoreover, the topic model augmentation can be applied to pretrained transformers. This makes it very versatile and greatly improves its usefulness in practice.\n\nHowever,\n\nThe presented work is motivated by the fact that transformers are limited in the length of the input they can condition on, and that topic information from previous segments could alleviate this problem. The model is then evaluated on GLUE, which is not a test of long-term dependencies. \nIt remains unclear whether providing topic information of preceding segments is enough to allow the model to draw information from these segments that is useful for a task, beyond mimicking their style.\n\nThe authors propose to use the learnt topics to perform conditioned topic-specific text generation. Given the multi-layer topic model architecture and the fact that GPT-2 uses BPE tokenization, splitting words into subwords, this raises the question whether most topics are interpretable enough to use them in a meaningful way for conditioning manually..\n\nMoreover, I do not find the results convincing that this methods works. I do not think that proper emphasis has been put on the baseline as it is diverging from its initial solution.\n\nIn general, I find this paper hard to read and keep track of what is being proposed. I find that it convolutes simple concepts and it could be a lot easier to understand the proposed method if written in a different way.\n\nI think the 54 citations in the introduction is unnecessary, please keep your paper more concise on what exactly you are building on top of and what you're proposing.\n\nI don't fully understand the TE embeddings. Is it N-gram distributions that you make a topic model over?\n\nIs WE a \"normal\" word embedding?\n\nPlease elaborate on E_A and E_B", "extra_scores": {}, "extra_text": {}}, {"review_id": "FYReshArlB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an interesting idea of enhancing the contextualised word embedding learned by Transformers with long-range semantic dependencies via topic learned by Poisson Gamma Belief Network (PGBN), a deep topic model. To leverage the topic information to guide the learning process of transformers, the authors proposed two types of topic-ware embeddings and one topic attention mechanism. The experimental results show incorporating topic information can further improve the performance of Transformers.\n\nOverall, it is an interesting paper. I would lean towards accepting it. I like the idea of injecting topic information into neural language models, like transformer. The ablation study shows that the performance of base models increases with the topic information.\n\nPros\n1.  Capturing longer-range term dependency in Transformers is an important problem. Although there are existing works on increasing the size of the input, such as Hierarchical transformers, the authors propose to used deep probabilistic topic model to leverage the semantic information via latent topics.\n2. The paper provides both quantitative results on both language generation and GLUE tasks and qualitative analysis, both show the necessity of considering topic information in transformers. The ablation study on the three components used to inject topic information is good\n3. The paper is written pretty well, and easy to read.\n\nCons:\n1. Although the ablation study is good already, I would suggest the authors conduct the following ablation studies wonder 1) how the regularisation terms would affect the performance? 2)  will the interleaving discussed on page 5 have an impact on performance?\n2. Figure 4 visualises the topic attention mechanism. What are those frequently attended topics?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes enhancing contextualized word embeddings learned by Transformers by modeling long-range dependencies via a deep topic model, using a Poisson Gamma Belief Network (PGBN). The experimental results show incorporating topic information can further improve the performance of Transformers. While this is an interesting idea, reviewers pointed out some weaknesses: 
- GLUE evaluation is not a test of long-term dependencies, it remains unclear whether providing topic information of preceding segments is enough to allow the model to draw information from these segments that is useful for a task.
- The improvement over the baseline does not seem to be significant.
- The ablation study could be improved and more experiments could be done to understand the effect of hyperparameters choices from the topic model, such as the number of layers of PGBN as well as the topic number of each layer.
- A comparison of the model performance for different lengths of input sequences would be helpful.
- There are many recent methods for long.range transformer transformer variants, it would be interesting to compare them against the proposed latent topic-based method.

Unfortunately, no answers are provided by the authors to the questions asked by the reviewers, which makes me recommend rejection.
Decision: Reject